Techniques for recognizing temporal tapping patterns input to a touch panel interface

ABSTRACT

Briefly, a method and apparatus for recognizing temporal tapping patterns input to a touch panel interface is disclosed. The method may include receiving user input with a touch panel interface, recognizing a temporal tapping pattern in the user input, and performing an action associated with the temporal tapping pattern.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application is a divisional application of U.S. patentapplication Ser. No. 12/641,585, filed 18 Dec. 2009, entitled,“MULTI-FEATURE INTERACTIVE TOUCH USER INTERFACE”, by Adamson et al.

BACKGROUND Description of the Related Art

In recent years, user interfaces to various electronic devices havebegun to provide some additional input capability. For example, touchscreen devices and soft-keyboards are becoming more common.

Touch-based interfaces provide a minimum amount of capability andflexibility, for example, only supporting simple one or two finger(s)touch-based gestures. For example, current interfaces only supportsimple key typing which activate the key on finger-down or finger-upactions or alternatively simple gestures like the two-finger zoom-in/outgesture and the single-finger drag gesture for scrolling. These devicestypically ignore or discard non-finger touches and require users to holdtheir hands above the surface unless they intend to type a key.

Further, written communication for text-based applications such as Emailis limited to the author's ability to convey their message with wordsand limited to the recipient's ability to extract the intended meaningwithout ambiguities or misunderstandings. Emoticons (such as a smiley:

) and a few text based acronyms (such as LOL) may be added to thecontent. These added features are limited to primitive smiley faces, use‘canned’ animation, express very general emotional ideas over the text,and further text features are not available directly through keyboardtyping. Although additional expressions in text based messages areavailable using today's defacto formats such as MicrosoftWord, they areunderutilized. The use of such expressions are limited due to thetediousness of constructing word-by-word or even character-by-characterfeatures for expressing the text message because these alterations areburied in a mouse and/or graphical menu driven paradigm that hinderspotentially flowing expression directly from our hands.

A more feature rich user interface is desired, one that minimizescomplexity and display cluttering while enhancing the user experience.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention may be better understood, and its numerousfeatures and advantages made apparent to those skilled in the art byreferencing the accompanying drawings.

FIG. 1 illustrates a basic electronic device with a feature rich touchsubsystem according to an embodiment of the present invention.

FIG. 2 illustrates a temporal tapping flow diagram of a touch subsystemcapable of receiving and interpreting temporal input signals accordingto an embodiment of the present invention.

FIG. 3 illustrates a debouncing of the onset of a non-finger touch shapeas performing a pattern initiation shape recognition according to anembodiment of the present invention.

FIG. 4 illustrates a touch screen view perspective of a temporal gestureaccording to an embodiment of the present invention.

FIG. 5 illustrates a typical tapped pattern tally according to anembodiment of the present invention.

FIG. 6 illustrates a representation of a time series derived from thetallied bins according to an embodiment of the present invention.

FIGS. 7A-7D illustrates an example of a multi-touch multi-shape gestureaccording to an embodiment of the present invention.

FIG. 8 illustrates a gesture identification flow diagram according to anembodiment of the present invention.

FIG. 9 illustrates a path created by a multi-touch, multi-shape gestureaccording to an embodiment of the present invention.

FIG. 10 illustrates several examples of such expressive touch gesturesaccording to an embodiment of the present invention.

FIG. 11 illustrates an expressive keyboard block diagram according to anembodiment of the present invention.

FIG. 12 illustrates a representation of a left hand in home position ona touch panel virtual keyboard according to an embodiment of the presentinvention.

FIGS. 13A-13C showcase three simple attributes, in particular, tilt,bold, and gentle text attributes, respectively, according to anembodiment of the present invention.

FIG. 14 illustrates a dual thumbs tracking position according to anembodiment of the present invention, with both thumbs on the surface inthe word attribute start position.

FIGS. 15A-15D illustrates subtle key animation attribute assignment withan index finger according to an embodiment of the present invention.

FIG. 16 illustrates a block diagram of a filtering unit according to anembodiment of the present invention.

FIGS. 17A-17H illustrates a touch panel sequence according to anembodiment of the present invention.

FIG. 18 illustrates a right hand brush with hover according to anembodiment of the present invention.

FIG. 19 illustrates a touch cell rate of intensity change over timeaccording to an embodiment of the present invention.

The use of the same reference symbols in different drawings indicatessimilar or identical items.

DESCRIPTION OF THE EMBODIMENT(S)

In the following description, numerous specific details are set forth.However, it is understood that embodiments of the invention may bepracticed without these specific details. In other instances, well-knownmethods, structures and techniques have not been shown in detail inorder not to obscure an understanding of this description.

References to “one embodiment,” “an embodiment,” “example embodiment,”“various embodiments,” etc., indicate that the embodiment(s) of theinvention so described may include a particular feature, structure, orcharacteristic, but not every embodiment necessarily includes theparticular feature, structure, or characteristic. Further, repeated useof the phrase “in one embodiment” does not necessarily refer to the sameembodiment, although it may.

As used herein, unless otherwise specified the use of the ordinaladjectives “first,” “second,” “third,” etc., to describe a commonobject, merely indicate that different instances of like objects arebeing referred to, and are not intended to imply that the objects sodescribed must be in a given sequence, either temporally, spatially, inranking, or in any other manner.

Unless specifically stated otherwise, as apparent from the followingdiscussions, it is appreciated that throughout the specificationdiscussions utilizing terms such as “processing,” “computing,”“calculating,” or the like, refer to the action and/or processes of acomputer or computing system, or similar electronic computing device,that manipulate and/or transform data represented as physical, such aselectronic, quantities into other data similarly represented as physicalquantities.

In a similar manner, the term “processor” may refer to any device orportion of a device that processes electronic data from registers and/ormemory to transform that electronic data into other electronic data thatmay be stored in registers and/or memory. A “computing platform” maycomprise one or more processors.

FIG. 1 illustrates a basic electronic device according to an embodimentof the present invention. Electronic device 100 may be any type ofelectronic device having a user interface, for example, including, butnot limited to personal computers, handheld devices, cell phones,consumer electronic device, kiosk, home appliances, and the like.Electronic device 100 may include a processing unit 102, memory 104,various I/O interfaces 106, a display 108, and a feature rich touchsubsystem 110. Processing unit 102 may include one or more processors orprocessing logic. Memory 104 may include a memory cache subsystem, adisk subsystem, bios memory, and/or other storage mediums. Various I/Ointerfaces 106 may include input and/or output subsystems such as audioI/O, a wired or wireless interface, a USB interface, and the like. Touchsubsystem 110 provides a feature rich user input and is furtherdescribed in the embodiments below. Note that one or more of theelements listed above may be omitted from electronic device 100, and inan alternative embodiment, touch subsystem 110 may be a stand-alonedevice.

According to embodiments of the present invention, feature rich touchsubsystem 110 includes one or more novel user input capabilities thatenhance the user experience of electronic device 100. Touch subsystem110 may be a large or small touch interface for receiving and/orinterpreting user input.

According to an embodiment of the present invention, touch subsystem 110is capable of receiving and interpreting temporal gesturing as userinput. A time-based tapping pattern may be used for quick and convenientaccess to applications and services. Such user input may require only asmall and/or dynamically-allocated region of a touch panel. According toan embodiment of the present invention, the user may tap a distinctpattern anywhere on the screen to activate a desired response, a methodideal for small displays with a limited amount of spatial real estate.

FIG. 2 illustrates a temporal tapping flow diagram of a touch subsystem110 capable of receiving and interpreting temporal input signalsaccording to an embodiment of the present invention. First, a patterninitiation shape is identified, block 202. This pattern initiation shapeis an indication that a temporal input pattern is about to begin. Thepattern initiation shape can be any unique pattern, for example, a threefinger touch in a location. Touch subsystem 110 stably detects a patterninitiation shape such as a distinct non-finger-shape (that is, anon-single-finger or other shape) interaction on the touch panel. Oncethe pattern initiation shape is identified, a persistent region may beestablished, block 204. A persistent region may be, for example, abounding box under the touched region for monitoring the subsequent tappattern. Monitoring for a tapping pattern may only be focused on theregion of the panel where the user placed the original patterninitiation shape. Touch up and touch down instances are monitored, timedand recorded, block 206. If, after the pattern initiation shape isidentified in block 202 or during the tallying of the touch up and touchdown instances in block 206, a touch down instance does not occur withina specified time threshold, the pattern times out, step 208. Therecorded pattern is normalized, block 210. Normalization of the patternis performed to compensate for user variations in pattern tapping speed.The pattern is compared to existing template patterns for correlation,block 212. Once a pattern is identified, the pattern is acted upon, ornegative results are reported and the persistent region removed, block214. The pattern may be associated with, for example the opening of aspecific application or some other preset computing event.

According to one embodiment of the present invention, if a tappingpattern is not recognized, the user may be prompted to associate thepattern with a computing event. Thus, patterns may be pre-defined ordefined during use by the user.

FIG. 3 illustrates a debouncing of the onset of a non-finger touch shapeas performing the pattern initiation shape recognition in Block 202 ofFIG. 2 according to an embodiment of the present invention. Asillustrated, the placement of a non-touchpoint shape on a touch panelevolves over several frames. It often starts with a single touchpointfollowed by other neighboring fingers as they land on the panel over aperiod of time, for example, multiple time windows of 50 msec or less.To properly classify the user's intention of placing a patterninitiation shape, for example, several clustered fingers on to thepanel, the debouncing stage waits a set amount of time, for example, 200msec after the onset of the initial touch observation before declaringthe interaction shape. If the shape remains a fingertip touch shapeafter this time, then no tapping is processed. If the shape of thecontact is determined to be the pattern initiation shape, for example, acluster of fingertips, then ‘tap start’ is declared and the processproceeds to Block 204 of FIG. 2. The method for shape classification maybe hull deformation between the cluster's hull and its perimeter as wellas the angle of its principle component analysis to the screen.

FIG. 4 illustrates a touch screen view perspective of a temporal gestureaccording to an embodiment of the present invention. Regions 402 through412 illustrate a sample tapping pattern over time. Region 402illustrates the pattern initiation shape action. The outlined box is thepersistent region which is established and monitored for a tappingpattern. Monitoring for a tapping pattern is only focused on the regionof the panel where the user placed the pattern initiation shape, forexample, a finger cluster depicted in Region 402. A box area under theclustered finger shape is defined and the detected pattern debounced.Regions 404, 408 and 412 illustrate touch up actions. Regions 406 and410 illustrated touch down actions. Region 412, a touch up action, timesout, ending the pattern and triggering the removal of the persistentregion. Upon time out, a tap sequence report will be generated and thebox region will stop being monitored for tapping patterns. For anothertap sequence to become active, another pattern initiation shape must bedebounced and recognized. Note that a tapping pattern may contain anynumber of touch down/touch up instances according to an embodiment ofthe present invention.

Touch up and down actions are monitored, timed and recorded. Timing maybe recorded as a number of frames that the touch up or the touch downaction was active until a touch up time out. A typical timeout of atouch up action (that is, no touch down actions within a period of time)may be 750 msec. In this way, a vector version of the sequence of everytouchup and touchdown is constructed in terms of incoming frame counts.The vector's contour represents the raw form of the temporal pattern ofthe user's sequence, and is eventually compared to existing patterns inblock 212.

FIG. 5 illustrates a typical tapped pattern tally according to anembodiment of the present invention. A horizontal ‘U’ indicates theaccumulated frames when no touchpoints are observed with in the region,and a ‘D’ represents a down touchpoint tallied when a contact is seenwithin the region. The upper ‘time out’ threshold is use to compare a Ucount to a maximum value. When any up U tally exceeds this threshold,the pattern is deemed complete and the recording is terminated andcompared to other patterns in memory.

After the pattern is tallied and timed, the pattern is normalized, block210. The histogrammed vector established in block 206 may be normalizedby converting all tallied bins as illustrated in FIG. 5 into floatingpoint values and then dividing each individual tally by the number oftotal frames used in the entire histogram. Each histogram bin (which isnow by definition a floating point value less than 1.0) may bemultiplied to 1000 to indicate a common 1000 sample time series forevery tapping pattern. Doing so guarantees that the entire sequencesoccurs in a fixed time scale for comparison with existing temporalpatterns.

Once normalized, the pattern is correlated with other templates orpatterns, block 212. From the normalized histogram vector, a time seriesmay be constructed where every touchup frame is represented by a −1 andevery touchdown frame is represented by a +1. This time series isestablished by transposing the histogram in a left to right fashion intoa 1000 point vector in memory of +−1 values. FIG. 6 illustrates arepresentation of a time series derived from the tallied bins accordingto an embodiment of the present invention. Other time series patterns,previously defined in the system, are then tested for closeness. Crosscorrelation is used between two normalized time series using +−2 framelags. Cross correlation for a single lag is performed using:

$\sum\limits_{n = 0}^{N - 1}{{T(n)}{{P\left( {n - l} \right)}/1000}}$

where T represent an existing serial pattern,

-   -   P represents the new recorded pattern    -   n represents the normalized sample count (1-1000)    -   and 1000 represents the normalization size of any one serialized        vector

The lag ‘1’ shown in the above equation is used to compensate for driftin the template by sweeping the ‘1’ from −2 to +2 in single unitincrements. This produces five correlation peaks computed near theoriginal patterns starting sequence. The outcome of this correlationfunction is +1 for a perfect match and −1 for a perfect mismatch.

Once the tapping pattern has timed out in block 208, the region isdisbanded, and the area is relinquished back to normal system touchinteraction, block 210. Confidence factors and pattern name may bereported in block 210. The maximum peak of the normalizedcross-correlation may be compared to a threshold of 0.8. If any of thecorrelation peaks (at any of the lags) exceeds this threshold, then themaximum is reported along with the established pattern name for thispattern. In other words, the largest correlation peak of any pattern T,on any lag ‘1’ is declared a pattern match to that template, with thelargest correlation peak of all existing template patterns is reportedas the best match for this user pattern.

According to one embodiment of the present invention, in the case of anunrecognized pattern, the user may be prompted to define the new patternor try again, block 210.

In the case where the user can not remember the pattern choices, thenthe system may in some way notify the user, for example, play an audiotone and/or show a dot pattern on the screen that mimics the patternover a 3 second duration. In this way the report and template patternsmay be shared and constructed with the user.

Currently, a user must access non-visible windows and platform featuresby rummaging through icons, menu bars, and feature lists. For specialplatform features such as extended displays, wireless activation, andscreen brightness, the OEM must add hardware buttons and take up surfacereal estate. According to an embodiment of the present invention, spacerequirements, overhead, and cost are removed and/or reduced.

According to an embodiment of the present invention, feature rich touchsubsystem 110 may be capable of receiving and recognizing gestures basedon combinations of finger and/or non-finger touches. Touch subsystem 100includes gesture identification techniques capable of identifyingdifferent multi-touch multi-shape gestures by analyzing temporal andspatial relationships between multiple touches according to anembodiment of the present invention.

A gesture may be pre-defined or defined by a user at run-time by theuser associating a specific action or command to the gesture. Auser-defined gesture may be used as a user password with added securitymeasures imposed or any other such desired action or command.

A gesture may be created by concurrent use of the fingers and/or palmsof one or both hands that occur in a single user interaction. A singleuser interaction may be defined from the start of a touch on the panelto the point when there is no touch on the panel. The start of a gesturemay be determined as either the start of the user interaction in apre-designated area of the panel or the start of a specific systemgesture that in turn signals the start of the multi-touch multi-shapegesture.

FIGS. 7A-7D illustrates an example of a multi-touch multi-shape gestureaccording to an embodiment of the present invention. The gesture ofFIGS. 7A-7D is created using a finger and the heel of the palm of onehand and the specific interaction created by their movement.

As illustrated in FIG. 7A, a pre-defined three finger gesture indicatesto the system to start recording a multi-touch gesture. Any fingerand/or non-finger (non-finger herein refers to non-single finger)combination may be defined as a gesture initialization and the presentinvention is not limited to the combination illustrated in FIG. 7A. Asillustrated in FIG. 7B, once the recording starts, the three fingerstransition to a single finger and the heel of the palm simultaneouslymakes panel contact. As illustrated in FIG. 7C, the finger then moves tothe right that also results in slight rotation of the heel of the palm.As illustrated in FIG. 7D, the hand then drags down causing the fingerand the heel of the palm to move down. And finally the hand is liftedoff the panel (not shown) causing the user interaction to end. Accordingto an embodiment of the present invention a gesture may be defined bythis entire user interaction from the start of gesture recording to theend of the interaction.

FIG. 8 illustrates a gesture identification flow diagram according to anembodiment of the present invention. Upon recognition of the start ofthe gesture (as illustrated in FIG. 7A), three main tasks are performed,features extraction block 802, followed by feature normalization block812, and then recognition block 822.

Features extraction block 802 extracts a set of features from thegesture-related user interaction. A multi-touch, multi-shape gesture ischaracterized by a set of touches, their evolution over time and theirspatial and temporal relationship with respect to each other. Featuresextraction block 802 includes multiple subtasks, blocks 804-810. Inblock 804, a region of touches is identified. A region can be defined asan individual touch or a predefined area on the panel or anautomatically generated area on the panel that encompasses a set oftouches that are grouped together based on proximity. In block 806,these regions are tracked over time for the duration of the gesture. Inblock 808, features from each region are extracted. Specific featuresthat can be extracted include the location (x,y) and shape (s) of theregion at each sampled time instant. An example of the shape classes maybe finger/non-finger classes that represent a shape created by a singlefinger touch and those created by non-finger regions of the hand, forexample, palm, edge or multiple fingers together. In block 810, a timeseries of the extracted features over the duration of the gesture iscreated. FIG. 9 illustrates a path created and the features of the tworegions of the example gesture in FIG. 7 according to an embodiment ofthe present invention. The time series may be illustrated as:

{(x_(t) ₁ ¹,y_(t) ₁ ¹,s_(t) ₁ ¹),(x_(t) ₁ ²,y_(t) ₁ ²,s_(t) ₁ ²), . . .,(x_(t) ₁ ^(M),y_(t) ₁ ^(M),s_(t) ₁ ^(M))}, {(x_(t) ₁ ¹,y_(t) ₁ ¹,s_(t)₁ ¹),(x_(t2) ²,y_(t) ₂ ²,s_(t) ₂ ²), . . . ,(x_(t) ₂ ^(M),y_(t) ₂^(M),s_(t) ₂ ^(M))}, . . . {(x_(t) _(N) ¹,y_(t) _(N) ¹,s_(t) _(N)¹),(x_(t) _(N) ²,y_(t) _(N) ²,s_(t) _(N) ²), . . . ,(x_(t) _(N)^(M),y_(t) _(N) ^(M),s_(t) _(N) ^(M))}where,

-   -   M: total number of regions    -   N: total number of time samples of this gesture    -   (x,y): the location of a region on the panel    -   s: the shape of the region

Once the features are extracted, there is a need to normalize thefeatures for later matching with the training templates of the gestures.Different examples of the same gesture will show variations due to theuser variations in speed of hand movement during gesture creation, whereon the panel the gesture is created, variations due to the angle of thepanel and also size of the gesture itself For user-dependent gestures,the size of the gesture represents a user-specific characteristic. Hencethe normalization may account for all the other variations besides sizeto preserve the user-specific information.

Feature normalization block 812 includes multiple subtasks, blocks814-818. In block 814, the path of the gesture is resampled to normalizefor speed variations during gesture creation. The speed manifests in thevariable number of samples of a gesture obtained during differentcreations of the same gesture. Resampling normalizes the gesture path toa fixed number of samples to enable matching with the training templatesof the gesture. In block 816, the gesture is normalized for locationvariation on the panel. The center of mass of the entire gesture path ismoved to (0,0) by removing the spatial mean of the width and height ofthe path. In block 818, the gesture is normalized for rotationvariations. The gesture path is anchored around its spatial center. Thepath can be rotated such that the starting sample of the gesture and thespatial center are aligned at 0 degree angle.

In recognition block 822, after gesture path normalization, the gestureis compared to the training templates of allowed gestures for gestureidentification. There are a variety of methods that can be used for thispurpose. A geometric distance measure is one such measure that iswell-suited for the purposes of user-defined gestures. User-definedgestures are limited by the amount of training data available and makeit difficult to create complex training models (such as Hidden MarkovModels) for recognition. The distance measure can be applied on thelocation and shape features of the normalized feature vector such thatthe spatial and temporal relations are compared. The gesture of thetemplate that results in the smallest distance can then be assigned tothe created gesture.

The above description assumes a path associated with the gesture. Forstationary gestures (gestures not involving any movement of thetouches), the recognition can be accomplished by shape identificationalgorithms according to an embodiment of the present invention.

According to an embodiment of the present invention, feature rich touchsubsystem 110 provides a convenient and intuitive method for users toconvey meaning and context within their text based messages. Accordingto an embodiment of the present invention, users may express messages intheir text using special touches on a touch screen, as the touch screenextracts additional context directly from the surface interaction. Theuser types on their keyboard to enter text into applications such ase-mail or blog front-end. The keyboard may be enhanced to interpret theuser's touch interacts with each key stroke. The keyboard may also beenhanced to interpret the user's touch interactions over the entiretyping surface. These interactions add additional attributes to theirmessage, for example, attributes that help create clarity and addedexpression beyond just the words. The user does not need to hunt forspecial menu items for colors and palettes currently buried in GUIdriven interfaces. Instead their typing ‘style’ and added ‘touchbehaviors’ generate a rich set of attributes on the fly. Theseattributes range from character-to-character font variations as well ashigher level text animation.

FIG. 10 illustrates several examples of such expressive touch gesturesaccording to an embodiment of the present invention. By tilting a fingerangle in 1002, the angle of the text is tilted. Holding fingers longeron a key in 1004, text is made bolder. By striking keys harder in 1006,text is made larger and/or bolder. By striking keys softer in 1008, textmay be smaller or a different font. By pinching and/or stretching a typephase in 1010, text may be space closer and or farther apart,respectively. How a user touches a phrase may add animation to the text(not shown.) There are many more possible expressive text optionspossible—embodiments of the present invention are not limited to thoseshown herein.

FIG. 11 illustrates an expressive keyboard block diagram according to anembodiment of the present invention. A traditional mechanical keyboardor a touch panel virtual keyboard 1102 may be used to receive userexpression information. For a touch panel virtual keyboard, the usertypes on a keyboard grid displayed on the horizontal surface of thetouch panel. The touching of the surface is captured with a touchsensitive technology such as projected capacitive sensing. The samplingrate should be large enough, for example, greater than 120 Hz, tocapture fast taps and sweeping movements on the surface. At least 4 bitsof dynamic range between light and firm presses of the surface should beprovided above the noise floor. The scanned area should cover thefingers, thumbs, and palm heels of two hands. The spatial resolution ofeach interaction should be at least 5 mm or if traditional mechanicalkeys are used, then tilt pressure on each key should be provided in lieuof the 5 mm spatial resolution.

FIG. 12 illustrates a representation of a left hand in home position ona touch panel virtual keyboard according to an embodiment of the presentinvention. Gray squares indicate a touch location on a multi-touchpanel. Lines 1202 and 1204 (eigen vectors of 2D areas) over the heel ofthe palm print and the dots 1206, 1208, 1210, and 1212 on the fingerprints indicate shape characteristics for determining hand part and keyattributes within a single frame. Shape attributes are tracked over time(not shown) for further interpretation.

Frame by frame touch objects are parsed along a variety of paths tofully describe their final presentation. Referring to FIG. 11, frames ofinteractions may be acquired within a touch surface scanner 1104 andstreamed in real-time to a shape classifier 1106. Shape classifier 1106assigns shape attributes to each surface interaction. In one embodiment,principle component analysis of the 2D shape and touch intensity may beused to assign the type of interaction and key attributes. Shapes areclustered into finger, thumb, and heel objects and passed along theirrespective paths.

A finger path classifier 1108 determines the keyboard key and the pathattributes for each struck character. For example, light touch, firmbrisk strike, sliding press, and long hold may be translated into fontcharacteristics of small thin, bold caps, elongated tilted, and boldtall fonts. Both a real-time object tracker and real-time Hidden MarkovModel (HMM) Viterbi search engine may be used to process path-dependentfinger movement that describe character and word attributes. FIGS.13A-13C showcase three simple attributes, in particular, tilt, bold, andgentle text attributes, respectively, according to an embodiment of thepresent invention.

A thumb-to-thumb path classifier 1110 adds additional characteristics towords and phrases beyond that conveyed by intentional finger touches.The initial position of the thumb for engaging these attributes is shownin FIG. 14. FIG. 14 illustrates a dual thumbs tracking positionaccording to an embodiment of the present invention, with both thumbs onthe surface in the word attribute start position. Using both thumbs, theword or phrase can be stretched, warped off the line, and reshaped asdesired. The markers for start/stop and the exact ‘thumb gesturelanguage’ may be determined by the application.

Heel shapes are passed to a heel state processor 1112 which interpretsthe heel position for further text attribute assignment. An example isshown in the first frame for FIG. 13A, where the left palm pad may beused to help determine that italics is intended for the next set of keystrokes. Touch area and ‘outer and inner’ pad closure are alsointerpreted for start and end of phrase attributes.

The interpretation of all of these shapes is passed to an attributegenerator 1114, which combines the various time synchronized surfacecharacteristics for a final attribute assignment of the keyboard stream.These attributes may include typical font attributes such as size,thickness, color, style, start of animation, end of animation, lineoffset positions, and motion paths.

Finally, an application translation layer 1116 is used to map thesecharacteristics to existing application interfaces including text statesof a vertical viewing screen.

FIGS. 15A-15D illustrate subtle key animation attribute assignment withan index finger according to an embodiment of the present invention.Shown are consecutive frames of the index finger sliding off the ‘d’ keyfrom left to right. This interaction may add a specific feature to thecharacter or word, for example, size animation.

According to an embodiment of the present invention, the keyboardcommunication channel may be used to embellish typed text with colorfuland shapely expressions of meaning conveyed from the way the user'shands touch the keyboard surface.

According to an embodiment of the present invention, a richinterpretation of contours made while typing on a touch sensitivesurface may be used to directly control a wide range of characterattributes. These attributes range from font and size to animated textstrings and pictures.

According to an embodiment of the present invention, feature rich touchsubsystem 110 includes a touch panel signal filtering method that candistinguish between a user's resting interactions and their fingertyping interactions. The filtering method provides a more naturalkeyboard response to the user's intentions than today's existing methodof simple palm signal rejection.

FIG. 16 illustrates a block diagram of a filtering unit according to anembodiment of the present invention. Filtering unit 1600 may include atouch sensitive panel layering with a visibly shown qwerty keyboard1602, a computation element 1604 to analyze touchtemporal/spatial/coupling information, a shape classifier 1608 todistinguish fingers from non-finger typing intentions, a proximityclustering unit 1610 to avoid transient panel reading from becomingfinger typing classifications, a touch duration threshold unit 1610 tolatch long panel interactions as rests and not typing strokes, anintensity change threshold unit 1612 for determining hover-brush vs.deliberate keystrokes, a distance traveled measurement unit 1614 todistinguish drags from taps, a decision logic 1616 to finalize a‘keystroke’ vs. ‘rest’ contact type, a rest override detector 1618 toprovide repeat-key characteristics, and a traditional keyboard interface1620.

FIGS. 17A-17H illustrate a touch panel sequence according to anembodiment of the present invention. The sequence details themulti-touch panel readings as a user types the letter ‘y’ on a qwertykeyboard from a resting position. Each figure is illustrated using aright hand from onset of the rest, to the key press, to lifting off thepanel. Data arrives into filtering unit 1600 in horizontal x, verticaly, and intensity readings. The x,y spacings may be 8 mm or smaller,providing a 2D matrix of 8 mm square intensities. A full matrix mayarrive into filtering unit 1600 at a rate of at least 30 per second,each matrix referred to as a frame. FIG. 17A illustrates a right handheel onset—note that this impression is similar to finger shapes. FIG.17B illustrates the next phase of a right hand heel onset—note theidentifiable shapes and emerging pinky. FIG. 17C illustrates a righthand in full rest. FIG. 17D illustrates a right hand in full rest withindex keystroke down. FIG. 17E illustrates aright hand in full rest withindex keystroke up. FIG. 17F illustrates a right hand initial lift. FIG.17G illustrates a right hand lift with ‘sticky’ transients. FIG. 17Hillustrates a right hand final lift.

Shape classifier 1606 latches shapes, for example, that are larger than14 mm diameter, as ‘non-touch’ shapes and removes them from the frame.The shapes may change from frame to frame as the user places their restmasses on the panel. As shown in FIGS. 17A-17H many transientinteractions appear as small fingertip shapes. Shape classifier 1606associates the largest shape seen across frames with the touch shapeeven if it later shrinks back to a ‘finger’ type shape. This avoidsfinger type shapes from becoming keystrokes on lift off—as seen in FIGS.17G and H.

Proximity clustering unit 1608 identifies all finger shapes that resideon the panel within 8 mm of non-finger shapes. This proximityobservation is associated with a shape across frames for the duration ofthe shape. This avoids transient finger shapes that often emerge on theedges of larger blobs, from being falsely released as keystrokes due totheir short panel existence. The touch shapes in FIGS. 17G and 17H thatexist on the panel for a short duration are excluded as keystrokesbecause of their proximity to other shapes.

All shapes are time-stamped at their onset by touch duration thresholdunit 1610. If a shape has remained on the panel for over 150 msec it isclassified as a resting contact. In one embodiment, this number may beset by the user. In another embodiment it is learned while the userinteracts with the keyboard. In FIGS. 17C-17F only the tap of the upperleft finger will fall below this threshold. All other contacts willexceed this time threshold and be latched as ‘Resting’.

All panel cell intensity values may be averaged with two different timeconstants by intensity change threshold unit 1612. A fast averaging maybe used which reacts to 50 msec contact onsets and a slow averaging maybe used which only reacts to 250 msec onsets. If the value spreadbetween the fast and slow averages exceed the ‘keystroke’ vs ‘hover’threshold, then a fast approach' is declared. FIG. 18 illustrates aright hand brush with hover (grey) according to an embodiment of thepresent invention. FIG. 19 illustrates a touch cell rate of intensitychange over time according to an embodiment of the present invention.Short lived panel brushes by the resting hand will be eliminated asnon-keystrokes by their slow contact onset.

The center of mass of a single shape is computed across frames and theEuclidean distance between the starting frame and ending frame iscomputed by measurement unit 1614. If the distance exceeds 8 mm, thenthis shape may be declared ‘Traveled’.

All shapes that have vanished from the panel with the followingassociated characteristics are declared a ‘Keystroke’ by keystrokedecision logic 1616: shape type ‘finger’ was sustained throughoutcontact; contact vanished from panel before Rest Duration exceeded;contact was not clustered with larger shapes; intensity onset rateexceeded hover threshold; center of mass did not travel on the panel;and center of the shape on the touch panel is declared a keystroke andmapped to one of the visual qwerty keys, according to an embodiment ofthe present invention.

To achieve the keyboard key-repeat behavior, the user double taps andholds the key down. Rest override detector 1618 monitors for this doublekeystroke and subsequent hold. If an area on the panel is struck andheld within 5 mm of its previous center of mass and within 300 msec,then the Rest Duration association is overridden, and the keystrokelocation is repeatedly delivered at a predetermined ‘repeat’ rate untilthe hold contact vanishes from the panel.

According to an embodiment of the present invention, the recognition oftemporal tapping by feature rich touch subsystem 110 solves displayclutter and poor screen utilization issues.

According to an embodiment of the present invention, temporal tappingmay be combined with path gestures. For example, a few taps and then adrag could be plausible. In addition, any combination of the previouslymentioned features may be used in feature rich touch subsystem 110according to an embodiment of the present invention.

The techniques described above may be embodied in a computer-readablemedium for configuring a computing system to execute the method. Thecomputer readable media may include, for example and without limitation,any number of the following: magnetic storage media including disk andtape storage media; optical storage media such as compact disk media(e.g., CD-ROM, CD-R, etc.) and digital video disk storage media;holographic memory; nonvolatile memory storage media includingsemiconductor-based memory units such as FLASH memory, EEPROM, EPROM,ROM; ferromagnetic digital memories; volatile storage media includingregisters, buffers or caches, main memory, RAM, etc.; and datatransmission media including permanent and intermittent computernetworks, point-to-point telecommunication equipment, carrier wavetransmission media, the Internet, just to name a few. Other new andvarious types of computer-readable media may be used to store and/ortransmit the software modules discussed herein. Computing systems may befound in many forms including but not limited to mainframes,minicomputers, servers, workstations, personal computers, notepads,personal digital assistants, various wireless devices and embeddedsystems, just to name a few. A typical computing system includes atleast one processing unit, associated memory and a number ofinput/output (I/O) devices. A computing system processes informationaccording to a program and produces resultant output information via I/Odevices.

Realizations in accordance with the present invention have beendescribed in the context of particular embodiments. These embodimentsare meant to be illustrative and not limiting. Many variations,modifications, additions, and improvements are possible. Accordingly,plural instances may be provided for components described herein as asingle instance. Boundaries between various components, operations anddata stores are somewhat arbitrary, and particular operations areillustrated in the context of specific illustrative configurations.Other allocations of functionality are envisioned and may fall withinthe scope of claims that follow. Finally, structures and functionalitypresented as discrete components in the various configurations may beimplemented as a combined structure or component. These and othervariations, modifications, additions, and improvements may fall withinthe scope of the invention as defined in the claims that follow.

1. A method comprising: receiving user input with a touch panelinterface; recognizing a temporal tapping pattern in the user input; andperforming an action associated with the temporal tapping pattern. 2.The method of claim 1, wherein recognizing the temporal tapping patternin the user input comprises: recognizing a pattern initiation shape;establishing a persistent region; tallying and timing touch up and touchdown instances into the temporal tapping pattern; identifying a patterntimeout in a touch up instance; normalizing a duration of the temporaltapping pattern; comparing the temporal tapping pattern to knownpatterns; and if the temporal tapping pattern is recognized as a knownpattern, performing the action associated to the known pattern.
 3. Themethod as recited in claim 1, wherein performing an action comprisesexecuting a complex action which otherwise would require one of multipleuser steps, a short cut action, and a macro function to execute.
 4. Themethod of claim 1, wherein the time-based tapping pattern is used forquick and convenient access to applications and services.
 5. The methodof claim 1, wherein the user input requires only a small and/ordynamically-allocated region of the touch panel.
 6. The method of claim1, wherein the touch panel interface is a screen associated with ahandheld device, cell phone or consumer electronic device and thetemporal tapping pattern is input anywhere on the screen to activate adesired response.
 7. The method of claim 2, wherein the persistentregion is a bounding box under a touched region for monitoringsubsequent tap patterns and wherein monitoring for a tapping pattern isonly be focused on a region where the user placed the pattern initiationshape.
 8. The method of claim 2, wherein once the pattern is identified,the pattern is acted upon, or negative results are reported and thepersistent region removed.
 9. The method of claim 2, wherein if atapping pattern is not recognized, the user is prompted to associate thepattern with a computing event and wherein patterns may be pre-definedor defined during use by the user.
 10. An apparatus, comprising: a touchsubsystem including a touch panel for receiving user input in the formof a temporal tapping pattern; and a processing unit to recognize thetemporal tapping pattern and perform an action associated with thetemporal tapping pattern.
 11. The apparatus of claim 10, whereinrecognizing the temporal tapping pattern in the user input comprises:recognizing a pattern initiation shape; establishing a persistentregion; tallying and timing touch up and touch down instances into thetemporal tapping pattern; identifying a pattern timeout in a touch upinstance; normalizing a duration of the temporal tapping pattern;comparing the temporal tapping pattern to known patterns; and if thetemporal tapping pattern is recognized as a known pattern, performingthe action associated to the known pattern.
 12. The apparatus of claim10, wherein executing a system action comprises executing a complexaction which otherwise would require one of multiple user steps, a shortcut action, and a macro function to execute.
 13. The apparatus of claim10, wherein the time-based tapping pattern is used for quick andconvenient access to applications and services.
 14. The apparatus ofclaim 10, wherein the user input requires only a small and/ordynamically-allocated region of the touch panel.
 15. The apparatus ofclaim 10, wherein the apparatus is a handheld device, cell phone orconsumer electronic device and the touch panel is a screen associatedwith the handheld device, cell phone or consumer electronic device andthe temporal tapping pattern is input anywhere on the screen to activatea desired response.
 16. The apparatus of claim 11, wherein thepersistent region is a bounding box under a touched region formonitoring subsequent tap patterns and wherein monitoring for a tappingpattern is only be focused on a region where the user placed the patterninitiation shape.
 17. The apparatus of claim 11, wherein once thepattern is identified, the pattern is acted upon, or negative resultsare reported and the persistent region removed.
 18. The apparatus ofclaim 11, wherein if a tapping pattern is not recognized, the user isprompted to associate the pattern with a computing event and whereinpatterns may be pre-defined or defined during use by the user.
 19. Anon-volatile computer readable medium encoded with computer executableinstructions, which when accessed, cause a machine to perform operationscomprising: receiving user input with a touch panel interface;recognizing a temporal tapping pattern in the user input; and performingan action associated with the temporal tapping pattern.
 20. Thenon-volatile computer readable medium encoded with computer executableinstructions of claim 19, wherein recognizing the temporal tappingpattern in the user input comprises: recognizing a pattern initiationshape; establishing a persistent region; tallying and timing touch upand touch down instances into the temporal tapping pattern; identifyinga pattern timeout in a touch up instance; normalizing a duration of thetemporal tapping pattern; comparing the temporal tapping pattern toknown patterns; and if the temporal tapping pattern is recognized as aknown pattern, performing the action associated to the known pattern.21. The non-volatile computer readable medium encoded with computerexecutable instructions of claim 19, wherein performing an actioncomprises executing a complex action which otherwise would require oneof multiple user steps, a short cut action, and a macro function toexecute.
 22. The non-volatile computer readable medium encoded withcomputer executable instructions of claim 19, wherein the time-basedtapping pattern is used for quick and convenient access to applicationsand services.
 23. The non-volatile computer readable medium encoded withcomputer executable instructions of claim 19, wherein the user inputrequires only a small and/or dynamically-allocated region of the touchpanel.
 24. The non-volatile computer readable medium encoded withcomputer executable instructions of claim 19, wherein the touch panelinterface is a screen associated with a handheld device, cell phone orconsumer electronic device and the temporal tapping pattern is inputanywhere on the screen to activate a desired response.
 25. Thenon-volatile computer readable medium encoded with computer executableinstructions of claim 20, wherein the persistent region is a boundingbox under a touched region for monitoring subsequent tap patterns andwherein monitoring for a tapping pattern is only be focused on a regionwhere the user placed the pattern initiation shape.
 26. The non-volatilecomputer readable medium encoded with computer executable instructionsof claim 20, wherein once the pattern is identified, the pattern isacted upon, or negative results are reported and the persistent regionremoved.
 27. A handheld device, comprising: a screen; a touch subsystemincluding a touch panel incorporated into said screen for receiving userinput in the form of a temporal tapping pattern; and a processing unitto recognize the temporal tapping pattern and perform an actionassociated with the temporal tapping pattern.
 28. The handheld device ofclaim 27, wherein recognizing the temporal tapping pattern in the userinput comprises: recognizing a pattern initiation shape; establishing apersistent region; tallying and timing touch up and touch down instancesinto the temporal tapping pattern; identifying a pattern timeout in atouch up instance; normalizing a duration of the temporal tappingpattern; comparing the temporal tapping pattern to known patterns; andif the temporal tapping pattern is recognized as a known pattern,performing the action associated to the known pattern.
 29. The handhelddevice of claim 27, wherein executing a system action comprisesexecuting a complex action by said handheld device which otherwise wouldrequire one of multiple user steps, a short cut action, and a macrofunction to execute.
 30. The handheld device of claim 27, wherein thetime-based tapping pattern is used for quick and convenient access toapplications and services associated with the handheld device.
 31. Thehandheld device of claim 30, wherein the temporal tapping pattern isinput anywhere on the screen to activate a desired response.